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Use of the Australian research assessment exercise, Excellence in Research for Australia (ERA) to influence the policy and practice of 
research education in Australia will undoubtedly have many consequences, some of them unintended and potentially deleterious. ERA is 
a retrospective measure of research quality; research education is prospective. There is a lack of alignment between the 2- and especially 
the 4- digit Fields of Research used for ERA and university organisational units. While numerous Fields of Research were rated as world 
class in multiple institutions in the capital cities of New South Wales, Victoria and Queensland, the other states and regional Australia have 
significant gaps. The Sciences, Technology, Engineering and Medical (STEM) fields were generally rated higher than the Humanities, 

Arts, and Social Sciences (HASS) disciplines. Thus using ERA results to allocate higher degree by research places will have highly variable 
consequences in different disciplines and locations, given the obstacles to the mobility of the largely mature-aged doctoral cohort and 
the forecast impending academic skills shortage. ERA provides an incentive for Australian academics to eschew publishing in low impact 
journals and is likely to disadvantage some research students for whom co-authorship in a lower impact journal is more advantageous 
than no publication. There are many ways in which ERA results could be used to improve the quality of research education in Australia. 
Nonetheless, simplistically limiting doctoral education to Fields of Research where an institution scored at or better than national or world 
averages in ERA is unlikely to be in the national interest because our future research and academic workforce needs to be well prepared to 
operate across the nation in areas of emerging research, including cross-disciplinary and applied research. 


Excellence in Research for Australia (ERA) is designed to 
provide a comprehensive review of the quality of research 
undertaken in Australian higher education institutions at 
regular intervals. The first ERA was conducted in 2010 
(Australian Research Council, 2011a), the second will 
be conducted in 2012 and the third is planned for 2016. 
ERA was a successor to the Research Quality Framework 


(RQF) (DEST, 2005); an initiative prompted by political 
scepticism about the claims/assertions that universities 
made about the value of and returns on national invest¬ 
ment in research. In implementing ERA, Australia follows 
several other countries, including the United Kingdom 
(RAE, 2008), New Zealand (PBRF, 2012), Hong Kong 
(French, Massy & Young 2001), which have conducted 
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national assessments of the quality of research based on 
various criteria.These overseas assessment exercises have 
been used to guide research funding in response to con¬ 
cerns about the affordability of funding all higher edu¬ 
cation institutions for research as higher education has 
moved from an elite to a mass system (Elton, 2000). How¬ 
ever, the outcomes have not always been as policy makers 
intended. For example, in the United Kingdom, the exer¬ 
cise, which was aimed at concentrating research in fewer 
institutions and departments, confirmed that many of the 
newer universities were producing quality research and 
many universities used their freedom of virement to fund 
lower-rated departments at the expense of higher-rated 
ones (Elton, 2000). 

In ERA 2010, each of the 41 Australian Higher Education 
Providers was invited to provide evidence of research 
quality, volume, application and esteem across eight dis¬ 
ciplinary clusters: (1) Physical, Chemical and Earth Sci¬ 
ences; (2) Humanities and Creative Arts; (3) Engineering 
and Environmental Sciences; (4) Social, Behavioural and 
Economic Sciences; (5) Mathematical, Information and 
Computing Sciences; (6) Biological Sciences and Tech¬ 
nology; (7) Biomedical and Clinical Health Sciences; (8) 
Public and Allied Health Sciences . The disciplines within 
each cluster were defined by the 2 and 4-digit Fields of 
Research identified by the Australian and New Zealand 
Standard Research Classification (ANZSRC, 2008). 

ERA 2010 was an academic rather than an end-user 
evaluation of Australia’s research. The evaluation was 
undertaken by eight Research Evaluation Committees, 
each of which was broadly representative of its discipline 
cluster group. Each committee’s assessment was based 
on a dashboard’ of indicators of research quality, research 
volume and activity, research applications and recogni¬ 
tion (Australian Research Council, 2011a). Each Field of 
Research was evaluated on a five-point scale ranging from 
1’ (well below world standard) to ‘5’ (well above world 
standard) with a rating of ‘3’ representing world standard. 
If an institution did not meet the low volume threshold 
for critical mass for a Field of Research, it was rated as 
not assessed’ for that field. The indicators were largely 
metric-based with an emphasis on citation analysis the 
vast majority of Sciences, Technology, Engineering and 
Medical (STEM) disciplines and peer review by interna¬ 
tional experts in the remaining discipline clusters. Thus 
the range of disciplines was split into peer-review disci¬ 
plines and citation disciplines. The evaluation processes 
were not transparent and attempts to determine the rela¬ 
tive importance of the input factors through retrospec¬ 
tive analysis have largely failed. Some bodies including the 


Australian Academy for the Technological Sciences and 
Engineering (ATSE, 2009) expressed concern that applied 
and cross-disciplinary research would be undervalued, a 
concern supported by analyses of British Research Assess¬ 
ment Exercises (e.g. Elton, 2000). 

The ways in which ERA will be incorporated into the 
drivers that determine the Research Training Scheme, the 
block grant provided to Australian universities to fund 
research training, have yet to be determined. In Research 
skills for an innovative future’ (DIISR, 2011a), the Austral¬ 
ian government stated that the Excellence in Research for 
Australia (ERA) initiative will support the ‘identification 
and recognition of research strengths within universities' 
as a vital component of research education (page 23). 
Despite intuitive appeal, this approach may have the unin¬ 
tended consequence of reducing research education in 
areas of national or regional importance, especially areas 
of applied, cross-disciplinary or emerging research. The 
purpose of our paper is to explore possible consequences 
of ERA for research education in Australia and to suggest 
ways in which ERA results could be used to enhance 
research education in Australia while minimising del¬ 
eterious, unintended consequences before they become 
apparent, let alone researchable’ (Elton, 2000). 

Methods 

Our analysis is largely based on the National Report of 
ERA 2010 (Australian Research Council, 2011a). ERA 
2010 scores were based on 25 2-digit and 157 4-digit 
Fields of Research as defined by the ANZSRC classifica¬ 
tion (ANZSRC 2008), a pragmatic taxonomy of research 
across all research and development sectors in Australia 
and New Zealand including industry, Government agen¬ 
cies, private not for profit organisations and universities. 
This classification was not designed as a taxonomy of 
university research per se and includes Fields of Research 
that are largely undertaken outside the sector e.g., auto¬ 
motive engineering and medical biotechnology. Thus it 
is questionable whether an analysis such as ours should 
include all these fields.Twenty-two of the 4-digit codes are 
‘XX99’ or ‘other’ codes e.g., 699 Other Biological Sciences 
and 1499 Other Economics. There were only 28 Units of 
Evaluation (a 2-digit or 4-digit Field of Research for one 
institution) across the 22 other’ Fields of Research com¬ 
pared with 1708 Units of Evaluation for the substantive 
Fields of Research (Commonwealth of Australia 2011a). 
The purpose of the other’ codes is to pick up research 
not adequately captured by the main 4-digit Fields of 
Research.Therefore, including these 22 Fields of Research 
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in an analysis of ERA distorts consideration of breadth, as a 
not assessed’ within these codes simply indicates there is 
adequate alignment of research codes and actual activity, 
whereas a ‘not assessed’ for a substantive code indicates 
that either there is no research activity at that Higher Edu¬ 
cation Provider, or if there is, it has not produced the req¬ 
uisite outputs to meet the threshold for assessment. 

There is also an argument that 1802 Maori Law should 
not be included in Australian assessments as the inclusion 
of this code in ANZSCR is a function of ANZSRC being 
a joint classification for Australia and New Zealand. No 
Higher Education Provider met the threshold for assess¬ 
ment for Maori Law in ERA 2010. 

In addition, nine 4-digit Fields of Research did not 
record any assessment. Whether that result indicates real 
gaps in the fabric of Australian Higher Education Research 
is beyond the scope of this paper.Thus ERA 2010 was not, 
in practice, an analysis of 157 4-digit Fields of Research but 
of 125 - 134 Fields of Research depending on whether the 
fields for which no returns were received are included.We 
used 134 Fields of Research in our analysis below by omit¬ 
ting the 22 other’ Fields of Research and Maori Law. 

Results and Discussion 

Challenges of ERA for research education 

Temporal scale mismatch 

ERA is a retrospective measure of research quality, volume, 
application and esteem aggregated into an overall per¬ 
formance rating. Based on data from eligible staff from 
each institution employed at the census date of 31 March 
2010, ERA 2010 applied to research outputs from 1 Janu¬ 
ary 2003 to 31 December 2008; research income, com¬ 
mercialisation and esteem measures between 1 January 
2006 and 31 December 2008; citation measures from 1 
January 2003 to 1 March 2010.Thus some of the research 
assessed must have predated the publications reference 
period by several years. The reference periods for ERA 
2012 will be updated, for example publications will be 
limited to the period 1 January 2005 - 31 December 2010, 
however, the exercise is inevitably retrospective. 

Most universities are investing in emerging areas of 
research to meet perceived future needs in the context 
of their institutional mission. Current doctoral candidates 
are the researchers of the future and their research should 
be aligned with research needs of the future rather than 
the research strengths of the past. Doctoral candidates 
should be well represented in an institution’s areas of 
emerging research including applied and cross-discipli¬ 


nary research. Experience in the United Kingdom sug¬ 
gests that these areas may not rate well (or at all) in ERA 
(Elton, 2000). 

Organisational scale mismatch 

There is a lack of alignment between the 2- and especially 
the 4- digit Fields of Research used for ERA and university 
organisational units. Most Australian universities are now 
organised in large multi-disciplinary schools that conduct 
research in many Fields of Research (e.g., Environmental 
Science staff at Griffith University contributed to 82 Fields 
of Research in ERA 2010 (Tony Shiel, pers comm 2011). 
Similarly at James Cook University, all of the assessed 
Fields of Research relied on inputs from at least two and 
typically five to eight of that institution's 25 academic 
organisational units (Chris Cocklin, pers comm 2011). 

In ERA 2010, this organisational scale mismatch was 
exacerbated by the inevitable attempt by every university 
to optimise its ERA returns. As a result, many staff, par¬ 
ticularly those undertaking cross-disciplinary research, 
contributed to their university’s return in several differ¬ 
ent Fields of Research, which may have received very 
different ERA evaluations. Alternatively, some institutions 
score well in Fields of Research not represented by their 
organisational units. For example, the Australian National 
University was rated as world class in Education at the 
2-digit level without having a unit in this discipline (Mar¬ 
garet Kiley pers comm. 2011). 

Although ERA 2012 will incorporate changes designed 
to improve the capacity to accommodate cross-discipli¬ 
nary research (Australian Research Council, 2011b), the 
changes are unlikely to improve this mismatch of organisa¬ 
tional scale.The revised methodology will allow each insti¬ 
tution to code journal articles with significant content (66 
per cent or greater) not represented by a journal’s Fields of 
Research to another appropriate Field of Research code of 
its choice (Australian Research Council, 2011b). However, 
institutions will still code publications to maximise their 
ERA scores rather than to align with organisational units. 
Thus using ERA results as a blunt instrument to define the 
fields, in which a university may offer doctorates or award 
Australian Postgraduate Awards for example, will almost 
certainly increase the perverse incentive to optimise' the 
coding of the Fields of Research in which research higher 
degree candidates are working, reducing the robustness of 
the data on this important topic. 

Perverse incentives 

ERA 2010 produced at least one perverse incentive 
that anecdotal evidence indicates has had an impact on 
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research training already. Because ERA 2010 was focused 
on all publications (or research outputs), it was perceived 
as emphasising publishing in highly ranked (A* andA) jour¬ 
nals in the case of the peer-review disciplines or journals 
with high impact factors in the case of citation disciplines. 
The Research Evaluation Committees were presented with 
percentages of A*, A, B and C publications in their dash¬ 
boards, along with other research indicators. Consequently 
until recently, some Australian academics were strongly 
encouraged to publish only in A* and A journals by senior 
university staff concerned that any publications in lower 
ranked journals inevitably reduced the percentages of 
publications in A* and A journals for the relevant ERA Unit 
of Evaluation. Thus some academics, particularly in the 
peer-review disciplines, perceived a strong disincentive to 
publish with a research higher degree candidate in a B or 
C journal. For the citation disciplines, there was a similar 
disincentive to publish in low impact journals. 

ERA 2012 will not use the controversial system of 
ranking journals used in ERA 2010 (Australian Research 
Council, 2011b). Rather the Australian Research Council 
will use a refined journal quality indicator and evaluation 
committees will use their expert judgement to assess the 
appropriateness of the journals for the disciplinary unit 
concerned. This new approach is less transparent than 
its predecessor and is unlikely to change the unwilling¬ 
ness of some supervisors to publish with their research 
students if it means publishing in low impact journals or 
their equivalent. 

Showing a research higher degree candidate how to 
publish is very much part of good practice in research 
training. Consequently, some doctoral programmes require 
all research students to publish a paper (or in some cases 
two papers) in order to satisfy the requirements for the 
degree. Research does not 
always work out as planned 
- there is an element of risk. 

When research does not work 
out or yields negative results, it 
is typically not possible to pub¬ 
lish the results in high impact 
journals. Tliis practice reflects 
the interest in the results to 
the readers of the journal, 
rather than the quality of the 
research. Journals are ranked 
on the basis of impact factor 
and it is inevitable that this 
information will be used in 
ERA 2012. Because ERA is cur¬ 


rently an assessment of all publications, any publication 
in a journal with a relatively low impact factor (includ¬ 
ing most journals in emerging fields and many journals 
that publish cross-disciplinary research) will still have the 
potential to dilute the quality of publications in the eyes 
of a Research Evaluation Committee. Thus many supervi¬ 
sors may be reluctant to publish in such journals with their 
research students, a practice that is likely to disadvantage 
the student. In addition, established journals can be quite 
conservative and reluctant to publish new work in emerg¬ 
ing, cross-disciplinary or applied areas. 

Systemic variables affecting the use of ERA in 
Research Education 

There are three broad variables associated with ERA out¬ 
comes that will have consequences if ERA is used to allocate 
higher degree by research places or government funded 
stipend scholarships: institutional grouping, geography and 
discipline.We consider each of these variables below. 

Institutional Grouping 

The performance of Australia's 41 Higher Education Pro¬ 
viders was predictably uneven in ERA 2010 (Table 1), 
although all but two universities were rated as at world 
class or better in at least one Field of Research indicating 
that as in the United Kingdom (Elton, 2000; RAE, 2008), 
some of the newer universities are producing some out¬ 
standing’ research (at least one university outside the 
Group of Eight universities achieved a maximum score in 
eight of the 18 2-digit Fields of Research). 

As expected, ERA confirmed the research standing of 
the Group of Eight universities which were collectively 
assessed in 692 Units of Evaluation of which 91-3 per cent 
were rated at world standard or better. The seven Innova- 


Table 1: ERA 2010 performance by institutional grouping 


Grouping 

# Institu¬ 
tions 

# Field of 
Research 
assessed 
/134 

# Units of 
Evaluation 
rated > 
world class 

# Units of 
Evaluation 
assessed 

% Units of 
Evaluation 
rated > 
world class 

Australian Technology 
Network 

5 

78 

134 

224 

59.8% 

Group of Eight 

8 

121 

632 

692 

91.3% 

Innovative Research 

Universities 

7 

95 

185 

296 

62.5% 

Non-aligned 

21 

98 

213 

496 

42.9% 

Regional 

14 

96 

189 

OO 

48.8% 

Total 

41 


1126 

1708 

65.9% 


Data Source: Australian Research Council (2011a). Excellence in Research for Australia 2010 National Report. 
Retrieved from www.arc.gov. au/era/. 
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Table 2: ERA 2010 performance by institutional state of origin 


State 

# Units of 
Evalua¬ 
tion rated 
> world 
class 

# Units of 
Evaluation 
rated 

% Units of 
Evalua¬ 
tion rated 
> world 
class 

# Fields of 
Research 

not 

evaluated 

# Fields of 
Research 
with only 

1 unit of 
evaluation 

# Fields of 
Research 
offered 
/134 

NSW 

359 

506 

70.9 

17 

19 

117 

VIC 

273 

395 

69.1 

22 

18 

112 

QLD 

205 

291 

70.4 

24 

29 

110 

WA 

107 

201 

53.2 

37 

38 

97 

SA 

107 

153 

69-9 

48 

38 

86 

TAS 

38 

54 

70.4 

80 

54 

54 

ACT 

65 

79 

82.3 

65 

59 

69 

NT 

6 

16 

37.3 

118 

16 

16 

Multi¬ 

state 

4 

13 

30.8 

n/a 

u/a 

n/a 

Total 

1164 

1708 

65.9 



134 


Note: Data Source: Australian Research Council (2011a). Excellence in Research for Australia 2010 
National Report. Retrievedfrom umo.arc.gov.au/era/. 


tive Research Universities collectively had 62.5 per cent 
of 296 Units of Evaluation rated at world class or better, 
a result similar to that of the five Australian Technology 
Network universities (59.8 per cent of 225 Units of Evalu¬ 
ation rated at world class or better). The performance of 
the 21 non-aligned institutions (42.9 per cent of 496 Units 
of Evaluation rated at world class or better), was more 
diverse, ranging from Macquarie with 75.6 per cent of 45 
Units of Evaluation rated as world class, to Batchelor Insti¬ 
tute of Indigenous Tertiary Education, University of Notre 
Dame and the University of the Sunshine Coast with none. 
The lowest performing 15 universities were assessed for 
234 Units of Evaluation although only 20.9 per cent of 
these were at world standard or better with the discipline 
of Nursing being the strongest performer with four of six 
universities being rated at or above world class in this 
Field of Research. 

Geography 

Geography matters. While New South Wales, Victoria and 
Queensland have numerous Units of Evaluation rated as 
world class in their capital cities, the other States (Table 
2) and regional Australia have significant gaps. South Aus¬ 
tralia does not have any institutions rated world class in 
two 2-digit Fields of Research: (1) Education and (2) Com¬ 
merce, Management, Tourism and Services. In the three 
4-digit Fields of Research in the discipline of Education, 
only two of the eight South Australian Units of Evaluation 
were rated as world class and only three of the nine Units 


of Evaluation across the seven 
4-digit codes in the Commerce 
cluster were considered world 
class. There are no world class 
providers in Western Australia in 
Law. There was only one institu¬ 
tion (Murdoch) rated at world 
class in Studies in Human Society 
at the 2-digit level and only three 
of 15 Units of Evaluation were 
rated as world class across the 
eight 4-digit Fields of Research in 
the Commerce discipline-cluster. 

We analysed the performance 
of 14 regional’ higher education 
providers; Ballarat, Batchelor, Cen¬ 
tral Queensland, Charles Darwin, 
Charles Sturt, Deakin,James Cook, 
Newcastle, New England, South¬ 
ern Cross, Southern Queens¬ 
land, Sunshine Coast, Tasmania 
and Wollongong. This grouping is a heterogeneous mix 
as it includes four institutions with no or one world- 
class ratings, three members of the Innovative Research 
University grouping (Charles Darwin, James Cook and 
Newcastle) while Tasmania and Wollongong are well- 
established non-aligned research universities. There were 
15 Fields of Research where the regional' universities 
scored relatively well, including Analytical Chemistry and 
Environmental Science and Management. Of the 33 world 
class Units of Evaluation across these 15 Fields, all but five 


Table 3: ERA 2010 results for the five 4-digit Fields of 
Research with the highest number of Units of Evalua¬ 
tion in 14 ‘regional’ institutions 


4-digit Fields of 
Research 

# Units of 
Evaluation 
rated > 
ivorld class 

# Units of 
Evaluation 

% Units of 
Evaluation 
rated > 
world class 

Curriculum & 
Pedagogy 

4 

14 

30.8 

Specialist Studies In 
Education 

3 

13 

23.1 

Education Systems 

1 

12 

8.3 

Sociology 

2 

11 

18.2 

Business & 
Management 

0 

13 

0.0 


Note: Data Source: Australian Research Council (2011a). Excellence in 
Research for Australia 2010 National Report. Retrievedfrom www.arc. 
gov.au/era/. 
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Table 4: ERA 2010: world class assessments for various Fields of Research 


2-digit Field of Research 

2-digit Fields of Research 

Aggregate of 4-digit Fields of Research per 
2-digit code 

# Units of 
Evaluation > 
world class 

# Units of 
Evaluation 

% Units of 
Evaluation > 
world class 

# Units of 
Evaluation > 
world class 

# Units of 
Evaluation 

% Units of 
Evaluation > 
world class 

Chemical Sciences 

26 

26 

100.0 

73 

84 

86.9 

Earth Sciences 

19 

21 

90.5 

57 

57 

100.0 

Agricultural & Vet Sciences 

22 

25 

88.0 

32 

36 

88.9 

Physical Sciences 

20 

24 

OO 

UO 

51 

60 

85.0 

Environmental Sciences 

20 

25 

80.0 

25 

30 

83.3 

Mathematical Sciences 

18 

24 

75.0 

50 

58 

86.2 

Engineering 

22 

31 

71.0 

96 

111 

86.5 

Built Environment & Design 

16 

23 

69.6 

32 

50 

64.0 

Biological Sciences 

23 

34 

67.6 

108 

132 

81.8 

Philosophy & Religious Studies 

16 

25 

64.0 

36 

50 

72.0 

History & Archaeology 

21 

33 

63.6 

33 

43 

76.7 

Medical & Health Sciences 

46 

73 

63.0 

184 

237 

77.6 

Information & Computing Sciences 

14 

24 

58.3 

21 

23 

91.3 

Language, Communication & Culture 

21 

36 

58.3 

66 

100 

66.0 

Technology 

4 

7 

57.1 

7 

8 

87.5 

Law and Legal Studies 

17 

35 

48.6 

17 

35 

48.6 

Psychology & Cognitive Sciences 

14 

31 

45.2 

20 

39 

51.3 

Education 

15 

39 

ir\ 

OO 

PT) 

42 

105 

40.0 

Studies In Creative Arts & Writing 

14 

38 

36.8 

63 

95 

66.3 

Economics 

12 

35 

34.3 

21 

49 

42.9 

Commerce, Management, Tourism 
& Services 

13 

39 

33-3 

60 

148 

40.5 

Studies In Human Society 

10 

38 

26.3 

70 

158 

44.3 

Total 

403 

686 

58.7 

1164 

1708 

68.1 


Note: Data Source: Australian Research Council (2011a). Excellence in Research for Australia 2010 National Report. Retrievedfrom www.arc.gov.au/era 


were located at the older institutions: Deakin James Cook, 
Newcastle,Tasmania orWollongong.The five 4-digit Fields 
of Research with the highest number of Units of Evalua¬ 
tion in regional institutions are listed in Table 3; only 10 of 
6l (16 per cent) Units of Evaluation were rated as world 
class or above. Hie result for Business and Management 
was particularly concerning: this Field of Research was 
not rated as world class at any of the 13 regional institu¬ 
tions that claimed critical mass. 

Discipline Matters 

One feature of ERA 2010 was the generally higher rating 
of the Sciences, Technology, Engineering and Medical 
(STEM) fields compared with the Humanities, Arts, and 


Social Sciences (HASS). The extent to which this result 
is an artefact of ERA methodology or reflects levels of 
maturity and/or investment in those fields is beyond our 
consideration. Thus using ERA results to allocate higher 
degree by research places will have highly variable conse¬ 
quences in different disciplines (Table 4). 

All Units of Evaluation were rated as world class or 
better for 40 (32 per cent) of 4-digit Fields of Research; 
66 Fields of Research (49 per cent) had >80 per cent of 
Units of Evaluation rated at world class or higher (Com¬ 
monwealth of Australia 201 la). For example, both Chemi¬ 
cal Sciences (100 per cent world class or better at the 
2-digit level) and Earth Sciences (100 per cent world class 
or better at 4-digit level), would be largely unaffected 
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by limiting higher degree 
by research students to 
institutions rated as world 
class in these disciplines. 

The alternative approach 
of limiting higher degree 
by research places to insti¬ 
tutions performing at or 
above national average in 
these disciplines would 
deprive world class groups 
of research students, policy 
that could not be in the 
national interest. 

However, less than half the Units of Evaluation were 
rated as world class for 18 Fields of Research, including 
some fields that were offered by numerous institutions: 
13 of these 18 low-rated Fields of Research were offered 
by between 27 and 39 institutions, one was offered by 22 
institutions and four were offered by between five and 
13 institutions (Australian Research Council, 201 la). The 
4-digit Fields of Research with the lowest percentage of 
world class ratings were Policy and Administration (18.5 
per cent - 27 Units of Evaluation), Marketing (27.6 per cent 

- 29 Units of Evaluation), Education Systems (31.3 per cent 

- 32 Units of Evaluation).Applied Economics (33.3 per cent 

- 33 Units of Evaluation), and Business and Management 
(33-3 per cent - 39 Units of Evaluation).Thus any mecha¬ 
nistic application of ERA to research education is likely to 


Table 6: Broad Field of Education and gender of Aus¬ 
tralian Doctorate by Research enrolments 2009 


Broad Field of Education 

Male 

Female 

Total 

Natural & Physical Sciences 

4693 

4470 

9163 

Information Technology 

1164 

438 

1602 

Engineering & related 
Technologies 

3825 

1222 

5047 

Architecture & Building 

339 

278 

617 

Agriculture, Environmental & 
related Studies 

1087 

1034 

2121 

Health 

2116 

3861 

5977 

Education 

1187 

2228 

3415 

Management & Commerce 

2049 

1557 

3606 

Society & Culture 

4515 

6366 

10881 

Creative Arts 

800 

1064 

1864 

Total 

21775 

22517 

44292 


Note: Data Source: http://www.deewr.gov.au/HigherEducation/ 
Publications/HEStatistics/Publications/Pages/2009FullYear.aspxAll 


significantly affect Economics, Commerce, Management, 
Tourism and Services and Studies in Human Society. 

Limiting access to Australian Postgraduate Awards to 
institutions scoring a world class ERA rating would clearly 
be problematic, especially as 61.9 per cent of doctoral 
candidates in 2009 were older than 30 (Table 5) and often 
have family arrangements that limit mobility. Although 
institutions could award university scholarships to doc¬ 
toral candidates in the disciplines in which they did not 
score well in ERA, this practice would reduce the attrac¬ 
tiveness of Australia to international research students 
because of the consequential reduction in the number of 
scholarships available to them. This approach would be 
counter-productive public policy because of the well doc¬ 
umented impending shortage of academics in Australia 
(Edwards, 2010; Edwards, Bexley & Richardson, 2011; 
Edwards, Radloff & Coates, 2009; Edwards & Smith, 2010; 
Hugo, 2008; Hugo & Morriss, 2010), the planned expan¬ 
sion of the sector (DEEWR, 2009; DIISR, 2009) and the 
increased international competition for the best and the 
brightest doctoral students. 

This problem is exemplified by the discipline of Edu¬ 
cation in which 3415 doctoral candidates were enrolled 
in 2009; 7.7 per cent of all Australian doctoral candidates 
(Table 6). Nearly 60 per cent of research students in Edu¬ 
cation surveyed in 2010 (Edwards, Bexley & Richardson, 
2011) were aged above 40, suggesting limited mobility. 
Only 15 of 39 institutions scored at or above the world 
average for the 2-digit Education Field of Research; no 
unit of evaluation received a maximum score. Thirty to 
50 per cent of the Units of Evaluation for each the four 
4-digt codes were also assessed at less than world aver¬ 
age (Table 7). Our comparison of the ERA 2010 data at 


Table 7: FRA 2010: results for Education Fields of 
Research 


Field of Research 

# Units of 
Evaluation 
> world 
class 

# Units of 
Evaluation 

% Units of 
Evaluation 
> world 
class 

Education (2-digit) 

15 

39 

38.5 

Education Systems 
(4-digit) 

10 

32 

31.3 

Curriculum & 
Pedagogy (4-digit) 

15 

37 

40.5 

Specialist Studies In 
Education (4-digit) 

17 

36 

47.2 


Note: Data Source: Australian Research Council (2011a). Excellence in 
Research for Australia 2010 National Report. Retrievedfrom www.arc. 
gov.au/era/. 


Table 5: Age distribution 
of Australian Doctorate by 
Research enrolments 2009 


Age 

Enrolments 

Under 19 

3 

20-24 

5213 

25-29 

11649 

30+ 

27427 

Total 

44292 


Data Source: http://www. 

deewrgov.au/HigherEducation/ 

Publications/HEStatistics/ 

Publications/Pages/2009FullYear. 

aspxAll 
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the 2-digit level and official higher education statistics 
purchased from the Australian government indicate that 
about one third of the total research students in Education 
were enrolled at institutions that were not rated as world 
class in ERA 2010, including 80 per cent of the domes¬ 
tic research students studying at regional institutions. 
Thus limiting research education in Education to institu¬ 
tions rated as world class at the 2-digit level will not only 
require the world class institutions to service a significant 
additional supervisory load (>1000 extra doctoral stu¬ 
dents) but would risk seriously downgrading Education 
research outside the mainland capital cities, particularly in 
Tasmania and regional Queensland. Given the importance 
of Australian educational practice being evidence-based 
and the impending shortage of academics in this field 
(64.9 per cent of staff are aged above 50; Edwards, Bexley 
& Richardson 2011), we consider that it is important to 
introduce mechanisms to promote high quality doctoral 
training in Education across the nation rather than to limit 
it based on past performance, a conclusion that we con¬ 
sider applies to many other disciplines as well. 

In ERA 2010, world-class critical mass was limited to 
five or fewer institutions in 39 4-digit Fields of Research 
(Australian Research Council 2011a). Nine 4-digit Fields of 
Research including Classical Physics had only one institu¬ 
tion with a world class ERA rating. Only seven institutions 
were rated as world class in Atomic, Molecular, Nuclear, 
Particle and Plasma Physics, a Field that is likely to be very 
important to Australia’s clean energy future and in which 
doctoral study should presumably be encouraged. 

To ensure that there was a ‘meaningful amount of data' 
to be evaluated, ERA 2010 had a low volume threshold 
for each Unit of Evaluation (Australian Research Council, 
2009). This threshold meant that an unknown number 
of isolated scholars' were not assessed, particularly in 
the Humanities where single scholars are the norm and 
in small institutions. There is anecdotal evidence that at 
least some of these scholars are very successful doctoral 
supervisors. Critical mass is very important in doctoral 
education to protect the interests of research higher 
degree candidates, especially if the principal supervisor 
becomes unavailable, institutional supervision using vir¬ 
tual technologies and visits is an increasingly-recognised 
practice, recently endorsed by changes to the Research 
Training Scheme to allow the recognition of joint comple¬ 
tions (DIISR, 2011b) We question the wisdom of exclud¬ 
ing high performing scholars who were not rated in ERA 
from research supervision and suggest that they should 
be encouraged to engage in cross-institutional supervi¬ 
sion as discussed further below. 


Possible solutions 

Changes to ERA to reduce the perverse student 
publication incentive 

A simple solution to overcome the negative impact of 
ERA on research student publications would be to require 
institutions to submit all publications (or research out¬ 
puts) as at present, but to present the data on only the top 
80 per cent of publications for each Unit of Evaluation 
to the Research Evaluation Committees. Such a change 
would enable supervisors to publish a less interesting 
paper with a research student in a low impact journal 
without a negative consequence when the relevant Unit 
of Evaluation is assessed for ERA. This reform could be 
introduced for ERA 2012. 

Using ERA to improve institutional practice in 
research education. 

The research environment is a necessary but not suf¬ 
ficient component of quality research education as 
acknowledged by the basket of indicators of doctoral 
training quality being developed by the Australian Council 
of Deans and Directors of Graduate Studies (Table 8). We 
consider that the planned revision of the Research Train¬ 
ing Scheme, the establishment of the Tertiary Education 
Quality Standards Agency (TEQSA), and the Compacts 
Process, together offer an opportunity for the Australian 
Government to require universities to explicitly take the 
quality of the research environment into account in devel¬ 
oping their policy and practices for research education 
and to audit their response. However, any policy change 
that uses the data from ERA should be designed to explic¬ 
itly address the challenges outlined above. 

Mission-based Compacts are three-year agreements that 
show how each university’s mission contributes to the 
Australian Government’s goals for higher education, and 
include details of major higher education and research 
funding and performance targets (DEEWR & DIISR, 2009). 
Requiring universities to stipulate how they plan to take 
their ERA results into account when awarding Australian 
Postgraduate Awards in their Compact Agreement and to 
audit this through the Tertiary Education Quality Stand¬ 
ards Agency would enable Higher Education Providers 
to respond in a more nuanced and positive way than if 
they were banned from awarding Australian Postgradu¬ 
ate Awards to doctoral candidates in Fields of Research 
that had been retrospectively evaluated by ERA as below 
world standard. Universities should also be able to iden¬ 
tify emerging Fields of Research that currently are not 
assessed’ or assessed below world standard, provide stra- 
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Table 8: Conceptual framework for the basket of indicators of the quality of 
research education developed by the Group of Eight chapter of the Australian 
Council of Deans and Directors of Graduate Studies. 

The relative importance of these indicators has not yet been debated and the Council acknowledges 


that the transaction cost of collecting the data for some indicators may be impracticably high. The 
indicators that could be informed by ERA are marked* 


Component 

Outputs 

Outcomes 

Graduates 

• Admission criteria 

• Student load 

• Number completed 

• Graduate Destinations 

• Student satisfaction 

• Completion rates/attrition 

• Employer satisfaction 

Contributions 
to knowledge 

• Thesis 

• Publications 

• Inventions 

• Exhibitions/major works 

• Examination outcome 

• Quality of Examination 

• Quality of Outputs 

Training 

programmes 

• Availability of Programmes 

• Programme Quality 

• Student satisfaction 

• Employer satisfaction 

Research 

education 

environment 

• Critical mass in areas of research strength* 

• Mentoring and supervising structures 

• Infrastructure for research and research 
education 

• International engagement 

• Interdisciplinary research experience 

• Student satisfaction 

• Research environment * 


tegic reasons why they wish to 
accept research higher degree can¬ 
didates or allocate Australian Post¬ 
graduate Awards to those Fields 
of Research, indicate how the 
research students will be provided 
with an appropriate research envi¬ 
ronment and negotiate how their 
performance should be evaluated. 

Several recent initiatives could 
be used in conjunction with ERA 
to improve doctoral education 
in fields of research in which 
there is a national or regional 
lack of critical mass. ERA offers 
a mechanism to identify such 
fields. Groups of universities can 
now share completions under the 
Research Training Scheme (DIISR, 

201 lb).Although this initiative has 
removed a significant barrier to 
cross-institutional co-operation in 
research education inAustralia, it is 
likely to provide a niche rather than an institutional solu¬ 
tion to the problems identified here. For example, over 
the last seven years, the Australian National University has 
developed several Memoranda of Understanding regard¬ 
ing joint PhDs with other Australian universities. To date, 
there has only been one cross-institutional PhD enrolment 
(Mandy Thomas, pers comm 2011) although recent Col¬ 
laborative Research Network agreements should improve 
this situation. Experience with developing joint degrees 
between Australian and overseas institutions indicate that 
the uptake and success of these arrangements is depend¬ 
ent on established individual collaborations rather than 
institutional Memoranda of Understanding. 

Institutional improvements to the quality of research 
education in a discipline could be achieved using struc¬ 
tures that are less formal than joint degrees: 

• Joint arrangements for embedded students with other 
providers e g. CSIRO. 

• Sharing of physical and virtual resources; 

• Incentives to encourage cross-institutional supervision 
and mentoring. 

• International collaboration with established research 
centres. 

• On-going collaboration (joint grants, papers, students, 
Collaborative Research Network agreements). 

• External input to milestones e g. Confirmation of Can¬ 
didature proposals. 


• External input into courses/skills development for 

research students. 

Nonetheless, such initiatives are likely to be expensive 
and need to be factored into the revision of the fund¬ 
ing for research training. In particular, research higher 
degree candidates may need assistance to travel between 
geographically separate institutions when distances are 
large, an inevitable feature of arrangements involving 
institutions in different states, especially the isolated 
regional institutions. 

Doctoral Training Centres are an increasingly-recog¬ 
nised approach to improving the quality of doctoral 
education by training cohorts of students while empha¬ 
sising transferable skills. In the United Kingdom, the Engi¬ 
neering and Physical Sciences Research Council and the 
Economic and Social Research Council have committed 
to fund more than 70 such centres, many in cross-disci¬ 
plinary and applied areas. The five Australian Technology 
Network (ATN) universities (only three of which were 
rated as world class in 2-digit Mathematics) have recently 
established a national Industry DoctoralTraining Centre in 
Mathematical Sciences (ATN, 2011) and its initial cohort 
of 20-25 PhD students will commence in early 2012, in 
nodes across the five ATN universities. Cross-institutional 
supervision can also be achieved using less formal struc¬ 
tures but research higher degree candidates will need 
travel assistance as explained above. 
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ERA as a block fund moderator 

The Australian government has indicated that the results 
of ERA will inform the allocation of funding to support 
the costs of research through the Sustainable Research 
Excellence Programme and research education through 
a modified Research Training Scheme.The Research Train¬ 
ing Scheme is the most valuable of the research block 
funding schemes, representing 41 per cent of the total 
allocation in 2011. For the Research Training Scheme, 
Australian Postgraduate Awards, and International Post¬ 
graduate Research Scheme, the calculation methodology 
(DIISR, 2011c) is relative institutional performance in 
research income (40 per cent), publications (10 per cent) 
and research student completions (50 per cent) and it 
expected that ERA results will be used to moderate these 
drivers. However at present, there is no agreed method 
of assessing overall institutional performance in ERA and 
some of the measures used are simplistic, including the 
measures such as percentage of Fields of Research at 
world standard or better used here. Of particular concern, 
especially for the large research intensive universities, is 
the failure of the present ERA rating scheme to include 
any measure of critical mass provided that the institu¬ 
tion meets the low volume threshold. An institution that 
achieves an ERA rating of‘5’ based on 50 publications in 
a Field of Research will provide a very different environ¬ 
ment for research high degree candidates to an institution 
that achieves the same rating based on 1000 publications. 
Nonetheless, bigger is not necessarily better, which is why 
a basket of indicators of research education quality is 
needed (Table 8). 

However, the volume of output that has gone into 
achieving an ERA rating has to be taken into account in 
the funding formula. It will be challenging to develop 
an agreed measure of overall institutional performance 
in ERA and use it to have a positive impact on research 
training while taking the following additional factors 
into account: (1) most universities in Australia produce 
some excellent research outputs as ERA 2010 demon¬ 
strated, (2) as in the United Kingdom (Elton, 2010), uni¬ 
versities are likely to use their freedom of virement to 
fund lower-rated department at the expense of higher¬ 
rated ones, (3) the challenges of Australia’s dispersed 
geography, (4) the impending shortage of academic staff 
identified by various scholars (Edwards, 2010; Edwards, 
Bexley & Richardson, 2011; Edwards, Radloff & Coates, 
2009; Edwards & Smith, 2010; Hugo, 2008; Hugo & Mor- 
riss, 2010), and (5) the need for quality academic staff to 
service the planned expansion of the sector (DEEWR, 
2009; DIISR, 2009). 


Conclusions 

ERA 2010 was a comprehensive academic evaluation of 
the research conducted by Australia's higher education 
providers in the first decade of the 21st century and 
subsequent rounds promise similar insights. Nonethe¬ 
less, use of ERA to influence the policy and practice of 
research education in Australia will undoubtedly have 
many unintended consequences, some potentially del¬ 
eterious. It is important to anticipate deleterious conse¬ 
quences before they become apparent. Our analysis of 
the results of ERA 2010 demonstrates a lack of alignment 
between the Fields of Research and university organisa¬ 
tional units and that using ERA results to allocate higher 
degree by research places will have variable conse¬ 
quences in different locations as a result of Australia’s 
geography and in different disciplines. In addition, ERA 
provides an incentive for Australian academics to eschew 
publishing in low impact journals, a practice which is 
likely to disadvantage some research students for whom 
co-authorship in a lower ranked journal is more advanta¬ 
geous than not publishing. 

Given these challenges, simplistically limiting doctoral 
education to Fields of Research where an institution 
scored at or better than national or world averages in ERA 
is unlikely to be in the national interest, especially given 
that ERA is retrospective and will not reflect the current 
situation. Doctoral students should be well represented in 
areas of emerging research including applied and cross- 
disciplinary research. 

There are many ways in which ERA results could be used 
to improve the quality of research education in Australia. 
We suggest that requiring Higher Education Providers to 
describe how they plan to deliver quality research edu¬ 
cation in all disciplines relevant to their mission in their 
Compact Agreement with the Commonwealth would be 
a positive reform. Institutions could also be required to 
report on their research education inputs and outcomes 
against an agreed basked of quality training indicators for 
each of these disciplines to the Tertiary Education Quality 
Standards Agency. 
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